Persistent Contextual Neural Networks for learning symbolic data sequences

We introduce persistent contextual neural networks (PCNNs) as a probabilistic model for learning symbolic data sequences, aimed at discovering complex algorithmic dependencies in the sequence. PCNNs are similar to recurrent neural networks but feature an architecture inspired by finite automata and a modified time evolution to better model memory effects. An effective training procedure using a gradient ascent in a metric inspired by Riemannian geometry is developed: this produces an algorithm independent from design choices such as the encoding of parameters and unit activities. This metric gradient ascent is designed to have an algorithmic cost close to backpropagation through time for sparsely connected networks. PCNNs are demonstrated to effectively capture a variety of complex algorithmic constraints on hard synthetic problems: basic block nesting as in context-free grammars (an important feature of natural languages, but difficult to learn), intersections of multiple independent Markovtype relations, or long-distance relationships such as the distant-XOR problem. On this problem, PCNNs perform better than more complex state-of-the-art algorithms. Thanks to the metric update, fewer gradient steps and training samples are needed: for instance, a generating model for sequences of the form ab can be learned from only 10 samples in under two minutes, even with n ranging in the thousands. This text is a preliminary version. The problem considered here is to learn a probabilistic model for an observed sequence of symbols (x0, . . . , xt, . . .) over a finite alphabet A. Such a model can be used for prediction, compression, or generalization. Hidden Markov models (HMMs) are frequently used in such a setting. However, the kind of algorithmic structures HMMs can represent is limited because of the underlying finite automaton structure. Below, we discuss examples of simple sequential data that cannot be, or cannot conveniently be, represented by HMMs; for instance, subsequence insertions, or intersections of multiple independent constraints.